Method of generating a patient-specific bone shell

ABSTRACT

The exemplary embodiments of the present disclosure are described and illustrated below to encompass methods and devices for designing patient specific prosthetic cutting jigs and, more specifically, to devices and methods for segmenting bone of the knee and the resulting cutting guides themselves. Moreover, the present disclosure relates to systems and methods for manufacturing customized surgical devices, more specifically, the present disclosure relates to automated systems and methods of arthroplasty cutting guides, systems and methods for image segmentation in generating computer models of knee joint.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. Non-Provisional patentapplication Ser. No. 13/203,010 which claims the benefit of U.S.Provisional Patent Application Ser. No. 61/208,509, filed Feb. 25, 2009and entitled, DEFORMABLE ARTICULATING TEMPLATE, and U.S. ProvisionalPatent Application Ser. No. 61/222,560, filed Jul. 2, 2009 and entitled,CUSTOMIZED ORTHOPAEDIC IMPLANTS AND RELATED METHODS, the disclosure ofeach of which is hereby incorporated by reference.

FIELD OF THE INVENTION

The present disclosure relates to systems and methods for manufacturingcustomized surgical devices, more specifically, the present disclosurerelates to automated systems and methods of arthroplasty cutting guides,systems and methods for image segmentation in generating computer modelsof knee joint.

INTRODUCTION TO THE INVENTION

The success of TKA operations depends on the restoration of correct kneealignment. It has been demonstrated that the key to the stability of theknee joint is the restoration of the mechanical axis with balancedflexion and extension gaps. Traditionally, intramedullary andextramedullary jigs were used to assist in orientation of the femoraland tibial components. Computer assisted surgery has been developed toassist surgeons in proper positioning and orientation of the components.However, surgical navigation systems are not widely used in hospitals.Main arguments against surgical navigation systems are the high costsand extra time spent in the operating room, in addition to a steeplearning curve.

The desire for a simple and accurate system as a substitute motivatedthe orthopaedic industry to develop a technique involving the use ofpatient specific cutting jigs. Magnetic resonance images (MRIs) of thepatient's hip, knee, and ankle are used to generate a model of specificpatient anatomy. From these images, software is used to create virtual3D models of the femur and tibia and orient these bones in space.Implant size is determined with the software and virtual bone resectionsare mapped to perform implant positioning and alignment using disposablecustom guides that fit on the patient's bone to determine pin placementfor the standard resection instrumentation of the implant manufacturer.

However, a number of drawbacks are associated with MRI includingscanning cost, increased scanning time, geometrical distortion, and theneed to standardize the scanning protocol among different MRI vendors.Recently Computerized Tomography CT is being used as well but theradiation associated with the procedure may not be favorable for somepatients. An alternative approach to MRI or CT is to use the statisticalanatomical shape analysis methodology to model the hylene cartilageaccurately using x-rays and/or ultrasound.

Statistical anatomical shape analysis has rapidly established itself asan invaluable tool in the design process for orthopaedic implants andpatient specific solutions. Thus, the intelligent Cartilage System (iCS)was conceived, with the goal of putting statistical anatomical shapeanalysis to accurately model cartilage thickness and contours to producecustom cutting guides (or Jigs). The guides are designed to place thecuts such that the knee is returned to its normal anatomical statebefore the degeneration of cartilage. The iCS system is built on thefoundational merger of statistics and three-dimensional bone modeling.

The iCS platform represents a cohesive software system encompassingmultidimensional medical imaging, computer aided design (CAD) andcomputer graphics features integrated for designing patient specificcutting jigs. The promise of coordinated interaction, efficiency, andmass customization allows iCS to address the complexities of increasedpatients' volume with accuracy and speed. A smart database composed ofbone and cartilage atlases provide technology utilized within thecustomized module for custom jig generation. By reducing turn-aroundtime in the current custom jig process, the iCS platform minimizes therisk of bottlenecks.

The database stores patient information following HIPPA regulations.Data from different imaging modalities will be attached with eachpatient including DICOM from MRI/CT, X-ray images, and ultrasound.Reconstructed bones and cartilage are stored in the database. Virtualtemplating data including calculated landmarks, axes, implant sizing,and placement information are also stored in the database. Thiscomponent implements both relational and XML schema to provide apowerful tool for data storage and manipulation.

Bone Cartilage Reconstruction Subsystem: This module involvesreconstruction of both bones and soft tissues either by segmentationfrom MRI, CT, PET or Ultrasound or direct reconstruction from RF signalsas in microwave imaging and US or three dimensional bone reconstructionfrom 2D X-ray images, flow chart outlining this subsystem can be foundin FIG. 4.

Segmentation of medical imaging can be roughly divided into twocategories; Structural and Statistical. Structural approaches are basedon spatial properties of the image pixels such as edges and regions.Statistical approaches rely on the probability distribution of intensityvalues in labeling the image regions. Image intensities and theircorresponding class labels are considered to be random variables.Therefore, the statistical approaches tend to solve the problem ofevaluating the class label, given the image intensity value. Previoussegmentation attempts have combined one or more of these methods tryingto overcome the limitations of individual methods.

Most of the segmentation using the edge oriented approach is semi-autosegmentation, where it is used to extract contours, optimizing contourand propagating it to neighbor slices. Image preprocessing and gradientoperators can be used to enhance typical region growing methods. If anorgan is known to be present in a certain area of an image by placementof a seed (either manually or automatically), the region can expanduntil it reaches the contrast boundaries of the organ. Parameters canvary to change the momentum of the growing, making the algorithm more orless sensitive to small changes in grayscale.

Region-Oriented Segmentation with Knowledge-Based Labeling is a pixelclassification technique based on the homogeneity of some features ofthe object. Various approaches such as knowledge based with uncertaintyreasoning, static domain knowledge from high order features extraction,fuzzy logic, long term and short term memory modeling, as well asunsupervised clustering have been used in this area. They yield verygood result with MRI images because of the high contrast between softtissues. The global interpretation error of the brain is 3.1% and theinterpretations error for sub-regions of the brain is 9%.

The watershed approach has been used in combination with various othersegmentation methods in hopes of improving accuracy. The watershedalgorithm is simply explained with a 3D plot of the grayscaleintensities in an image; the “water” fills the valleys of the imageuntil two valleys meet. This provides connectivity information aboutdifferent areas of the image by relying on grayscale values.Pre-processing with edge enhancement such as Sobel filters and textureanalysis can aid in detection of different organs.

Clustering algorithms are non-supervised algorithms that iterate betweensegmenting the image and characterizing the properties of each classuntil well defined image clusters are formed. Examples for suchalgorithms are the K-means algorithm, the fuzzy c-means algorithm, theexpectation maximization (EM) algorithm and Self-Organized NeuralNetworks. The K-means clustering algorithm clusters data by iterativelycomputing a mean intensity for each class and segmenting the image byclassifying each pixel in the class with the closest mean. It was usedto segment the brain. The number of classes was assumed to be three,representing cerebrospinal fluid, gray matter, and white matter.

The fuzzy c-means algorithm generalizes the K-means algorithm, allowingfor soft segmentations based on fuzzy set theory. The EM algorithmapplies the same clustering principles with the assumption that the datafollows Gaussian mixture model. It iterates between computing theposterior probabilities and computing maximum likelihood estimates ofthe means, covariances, and mixing coefficients of the model. Sincealgorithms do not directly incorporate spatial modeling they aresensitive to noise and intensity inhomogeneities. This can be overcomeby using Markov Random Field modeling.

Morphological Operators like binary morphology has been used in severalsegmentation systems, the basic idea in morphology is to convolve animage with a given mask (known as the structuring element), and tobinarize the result of the convolution using a given function. Examplesare: Erosion, Dilation, Opening and Closing.

Statistical approaches like thresholding label regions of the image byusing its intensity value histogram. A maximum and minimum thresholddefines the region on interest depending on the knowledge base. Forexample, in CT, rough segmentation of organs can be achieved bythresholding the image according to the Hounsfield unit ranges fororgans of interest. It has been applied in digital mammography, in whichtwo classes of tissue are typically present; healthy and tumorous.Limitations of depending solely on such a method are due to the usualoverlap of organ intensities' intervals, intensity inhomogeneities, andsensitivity to noise and image artifacts. Since thresholding does nottake into account the spatial characteristics of the image, any artifactthat distorts the image histogram can eventually affect the segmentationresults. Nevertheless, thresholding remains to be an initial step inmany segmentation algorithms Variations on classical thresholding havebeen proposed for medical image segmentation that incorporateinformation based on local intensities and connectivity.

Deformable template matching is performed with 2D deformable contours,which involves detection of the contour, tracking, matching andoptimizing the error of the match. In segmentation, 2D deformablecontour is applied with an atlas, which permits mapping between theimage data and the atlas by constrained minimization of predefined data.A 3D deformable surface is also realized by tracking contour changesbetween slides. Bayesian statistics is commonly used for determining themodel priors or likelihood in order to separate the organ of interestwith its neighboring organs. In general, the approach produces goodresults for small and local shape changes. It is also suitable for largeand global misalignments or deformations. A major improvement came fromusing principle geodesic analysis, which determines the mean shape of anobject and the principle mode of variation, and an m-rep model, whichthe object is represented as a set of connected meshes of medial atoms.This method yields excellent results for automatic segmentation ofkidneys with 0.12 cm mean surface separation and 88.8% volume overlay ascompare to manual segmentation.

Knowledge-Based Approach with Blackboard Architecture system, ingeneral, is an area of shared memory that contains a problem to besolved and a number of different processes. The blackboard iscontinually constructed and updated along the reasoning process. Thelabeling of the anatomical structures of the image data is done bymatching the structure in the image to the corresponding objects in themodels. The data from the image and the model are transformed into acommon, parametric feature space for comparisons. The low and high levelfeatures extracted are written to the blackboard and compare with themodel. The result is also written onto the blackboard to guide furthermatching. Long-term memory on the descriptions and relationships of theobject can be written into the knowledge base.

Four Dimensional Markov Random Fields (MRF), present a method using a 4Dprobabilistic atlases to segment moving targets such as the heart. Theatlases predict the time and space variance based on a prioriinformation; the algorithm also incorporates spatial and temporalcontextual information using 4D Markov Random Fields (MRF). Globalconnectivity filters finish the segmentation, using the largestconnected structure as a starting point. The results presented were veryfavorable for segmentation of the heart and are not limited to MRIimages. The results for left ventricle (LV) 96%, myocardium 92% andright ventricle (RV) 92% as compared to manually segmented models.

Our system utilizes the information from any three dimensional imagingmodality to extract gradients information combined with statisticalatlases to extract bone boundaries, and cartilage interfaces. Bellowdetailed description of each these bone reconstruction modalities.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of an exemplary iCS system overview.

FIG. 2 is a diagram of an exemplary data upload subsystem as shown inFIG. 1.

FIG. 3 is an exemplary database schema as shown in FIG. 1.

FIG. 4A and FIG. 4B are an exemplary listing of database table detailsfor the database shown in FIG. 1.

FIG. 5 is an exemplary schematic diagram of a bone/cartilagereconstruction subsystem as shown in FIG. 1.

FIG. 6 is an exemplary diagram of a virtual templating subsystem asshown in FIG. 1.

FIG. 7 is an exemplary diagram of a jig prototyping subsystem as shownin FIG. 1.

FIG. 8 is an exemplary diagram showing the processes of the algorithmfor automatic segmentation within the segmentation process in accordancewith the instant disclosure.

FIG. 9 is an exemplary diagram of the processes of an automaticalignment algorithm in accordance with the instant disclosure.

FIG. 10 is an illustration showing calculated crest lines as an exampleof the feature extracted from mesh for a distal femur.

FIG. 11 is an illustration showing profile search along normaldirections for CT.

FIG. 12 is an illustration showing profile search along normaldirections for MRI.

FIG. 13 is a graph showing the profile at various stages and the newedge location relative to the old edge location.

FIG. 14 is a segmented image with edge relaxation.

FIG. 15 is a segmented image.

FIG. 16 is a segmented image before relaxation.

FIG. 17 is a segmented image after relaxation.

FIG. 18 is an exemplary diagram showing the process of cartilagesegmentation from MRI.

FIG. 19 is a CT image taken with a contrast agent that allowsvisualization of cartilage tissue.

FIG. 20A and FIG. 20B are images of two femoral surfaces for profilecomputation, with image (a) including a tibia contact surface and image(b) including a tibia noncontact surface.

FIG. 21A, FIG. 21B, and FIG. 21C are mean profile graphs for class 1(a), class 2 (b) and class 3 (c).

FIG. 22 is an image of a segmented bone and cartilage from MRI.

FIG. 23 is an exemplary diagram of an X-Ray 3D model reconstructionprocess flow.

FIG. 24 is an image of an exemplary calibration target.

FIG. 25 is an exemplary image showing beads as would appear on aradiographic image.

FIG. 26 is a radiograph of the leg with the calibration target attached.

FIG. 27 are segmented images of a distal femur.

FIG. 28 is an image showing a complex pose search space and how particlefilters succeed in finding the optimum pose.

FIG. 29A and FIG. 29B are images showing the template bone's projectionon the radiographs.

FIG. 30 are images of contours mapped to 3D.

FIG. 31 is an exemplary diagram showing a 3D reconstruction process.

FIG. 32 is an exemplary diagram for training the prediction model forcartilage thickness.

FIG. 33 is an exemplary diagram for cartilage reconstruction usingtrained prediction model.

FIG. 34 are images of estimated cartilage thickness from MRI.

FIG. 35 is an exemplary cartilage template thickness map.

FIG. 36 is an exemplary image of predicted cartilage on a femur andtibia.

FIG. 37 is an exemplary diagram of a UWB imaging system.

FIG. 38 is an exemplary diagram showing how one signal acts as thetransmitter while signals reflected from the knee are received by all ofthe other UWB antennas.

FIG. 39 is an image showing an experimental setup where the UWB antennaarray surrounds the circumference of the knee.

FIG. 40 is a diagram for a microwave imaging process for detection oftissue interfaces at the femur and tibia.

FIG. 41 are exemplary samples of registered ultrasound images acquiredfrom an anterior distal femur.

FIG. 42 is an exemplary image of a bone model fit to acquired ultrasoundimages of the distal femur.

FIG. 43 is a diagram depicting the steps taken to segment usingultrasound.

FIG. 44 is a screen shot of a virtual templating subcomponent.

FIG. 45 is an exemplary diagram of a jig creation process.

FIG. 46 are sequential images representing certain steps of creating ajig.

FIG. 47A, FIG. 47B, FIG. 47C, and FIG. 47D are a series of imagesrepresenting different jig designs with different fixation (a medial andlateral condyle fixation, b curvature fixation, C groove fixation).

FIG. 48A, FIG. 48B, FIGS. 48C, and 48D are a series of images showing afemoral and tibial cutting jig for use in a knee revision surgicalprocedure.

FIG. 49 is a screen shot of a CAD editor for modifying the 3D output jigmodel.

FIG. 50 is a screen shot of an evaluation of a jig with respect tooriginal CT data.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

The exemplary embodiments of the present disclosure are described andillustrated below to encompass methods and devices for designing patientspecific prosthetic cutting jigs and, more specifically, to devices andmethods for segmenting bone of the knee and the resulting cutting guidesthemselves. Of course, it will be apparent to those of ordinary skill inthe art that the preferred embodiments discussed below are exemplary innature and may be reconfigured without departing from the scope andspirit of the present invention. However, for clarity and precision, theexemplary embodiments as discussed below may include optional steps,methods, and features that one of ordinary skill should recognize as notbeing a requisite to fall within the scope of the present invention.

Referring to FIG. 1, an outline of an exemplary overall system includesa surgeon creating a new case and requesting a custom jig and uploadingthe patient imaging data, this is followed by system notifying companyengineers about the new case. Next step involves creating patientspecific bone and cartilage which are then used to find implant the bestfit the patient, upon completion of preoperative planning surgeon isnotified to review and approve the planning. Once surgeon approves theplanning a custom cutting jig is automatically created for patient thattranslates the preoperative planning into the operating room.

FIGS. 2-7 outline the main components of the system this includes a dataupload component, database component, bone cartilage reconstructioncomponent, virtual templating and jig generation component.

Automatic segmentation process is outline in FIG. 8. First step insegmentation process is aligning base mesh from statistical atlas withthe volume an automatic alignment algorithm was developed to perform anaccurate alignment. This alignment process 3.a.1 is outlined in FIG. 9.This process involves extracting isosurface via simple thresholding. Theisosurface is basically a surface mesh generated from all of thebone-like tissue in the volume. The isosurface is noisy, and one can'tdistinguish separate bones. Extract features from the isosurface and themean atlas model (this can be done earlier and simply loaded). Thesefeatures could be, but are not limited to, crease, or crest lines FIG.10, umbilical points, or any other surface descriptor. Match featurepoints on mean model to those on isosurface via nearest neighbor orother matching method. The results here will be noisy, meaning therewill be several mismatches, but a subset will be correct. Using somerobust fitting method, for example, the RANSAC algorithm or the leastmedian of squares method, find the transformation that minimizes the en⁻or between matched points while maximizing the number of matches.

Upon completion of the previous alignment step an iterative warpingprocedure which uses the information in the atlas as a constraint on themodel's deformation is performed. The initial parameters determine thenumber of principal components to start with, the initial search lengthand the minimum allowed search length. The first step is to compute thevertex normals for each vertex on the bone mesh. These normal directionsrepresent the direction of deformation for each vertex. These normalsare calculated by averaging the normals of the adjacent triangles in themesh. The search line for each vertex, defined by the normal direction,is the path extending inward and outward from the bone model, centeredat the vertex. The length of the search line decreases as the processprogresses.

The intensity values for the search line are computed via trilinearinterpolation, as the sampling rate of the search line can be higherthan the given volume resolution. Profiles are first smoothed via somedenoising filter, here we use Savitsky-Golay filtering. This isespecially important given the noisy nature of MRI images. Afterdenoising, the gradient along the profile is calculated. This initialgradient is insufficient to determine bone edge location as there areseveral tissue interfaces with strong gradients, such as the skin-airinterface. This can be seen in FIGS. 11 and 12. As the initial automaticalignment is considered very accurate, it is safe to assume that thepatient-specific bone edge is located near to the aligned vertex. Tomodel this assumption, the gradient profile is weighted by a Gaussianfunction, so that the center vertex is given an initial weight of 1.0,with the weights decreasing as the search proceeds farther from thecentral position. An illustration of the profile at various stages canbe seen in FIG. 13. After the weighting, the absolute maximum of thegradient is determined, along with its location. This locationrepresents the bone edge; the old vertex location is then replaced withthe new edge location.

For CT the process seeks locations of falling edges, or edges that gofrom high intensity to low intensity as the search moves from inside tooutside. The opposite is true for the MR case. Therefore, if necessary,the profiles are flipped before the steps above to account for modalitydifferences. After the deformation for each vertex is performed themodel is aligned with the atlas using the inverse of the transformationcalculated in the initial alignment step. This model can be considerednoisy, as some edge locations are located at incorrect positions. Inorder to constrain the deformation and remove as many noisy points aspossible, we project the noisy model's vertices onto the atlas spaceusing a specific number of principal components, which is determined byparameter that change based on the residual error as in FIG. 14. Theresulting model will be a healthy femur which best represents thepatient specific noisy model. These models are then transformed back tothe volume space.

After projection, the parameters are updated, so that the search lengthis decreased, unless a new principal component is added. Then the searchlength is slightly increased so that new local information can becaptured. A new principal component is added when the residual RMSreaches a sufficiently low threshold. The process above, starting withthe normal direction calculation, repeats until all principal componentsare used and the residual RMS is sufficiently low, or until a setmaximum number of iterations is reached. The resulting noisy model,contains the patient-specific information, is then smoothed and remeshedat a higher resolution[FIG. 15,16]. The higher resolution allows thecapture of small local deformities, such as osteophytes, which may havebeen missed in the lower resolution segmentation process. The highresolution model is then relaxed by performing one iteration of thesegmentation procedure, using a sufficiently small search length toprevent incorrect bone locations, stopping before the bone is projectedonto the atlas. This resulting bone is a highly accurate representationof the patient anatomy. The output of the segmentation is then a highresolution patient-specific bone model and a smooth model,representative of the nearest healthy bone generated by the atlas beforerelaxation[FIG. 17].

After the automatic segmentation is complete we have 2 bones. The boneprojected on the atlas, and the relaxed, patient-specific bone. If thebone is highly degenerative, we can perform an unconstrained relaxationusing Gradient Vector Flow (GVF) snakes. These snakes respond to thegradient information present in the image and the flow of the gradientact as forces which cause the snake contour to locally expand orcontract to the boundary that minimizes the forces on the contour. Ifthe contour is not highly degenerative, then the initial relaxation stepmost likely is very near the actual patient anatomy and the snake methodis not necessary. The snakes and interactive segmentation tools can beseen in FIGS. 7 and 8. The final segmentation step before modelgeneration is an interactive intervention to correct any errors in thesegmentation process. Several tools are provided for interactivesegmentation, such as contour adding or subtracting, painting, etc. Oncethe segmentation has been confirmed, the final model is generated byinterpolating the contours at a fine resolution to ensure a smoothresult.

The process of cartilage segmentation from MRI is shown in FIG. 18, uponpatient volume segmentation, the resulting patient specific bone modelsof the distal femur, proximal tibia and patella can be used to acquirepatient specific cartilage models. If enough information is present inthe scan, which is the case when MRI is used or CT with a contrast agent(FIG. 19) that highlights the cartilage tissue, the cartilage can besegmented by utilizing a priori information along with measured patientspecific information. The a priori information can be considered as afeature vector.

In the case of the cartilage this can be thickness or locationinformation relative to the joint bones. The confidence placed in the apriori data is represented via the probabilistic models of each feature.For example, the confidence that the cartilage is x mm thick at acertain point on the bone is modeled in the cartilage thickness mapbuilt from previously segmented cartilage models. To determine theposterior probability, say, the cartilage thickness at a new point, aBayesian inference model is used, of the form

${p\left( x \middle| m \right)} = \frac{{p_{pr}(x)}{p\left( m \middle| x \right)}}{p(m)}$

Here, p(x|m) is the posterior probability given m, the measurements. Thevalue p(m|x) is the likelihood function representing the likelihood thata given measurement, m, would occur given a value of x. The p(m) term isa normalization term. The prior probabilities are present in ppr(x), theprior probability density. The best guess one can make for the value ofx given some measurement in is then the x which maximizes the posterioriprobability. This is known as the maximum a posteriori estimate (MAP).For cartilage thickness estimation (x), the MAP estimate is sought giventhe joint spacing (m) and the prior thickness map (ppr(x)). This sameconcept can be applied for BCI location.

Initially, the search is limited to the articulating surface. This isdone by utilizing the a priori information present in the atlas. Thecontact surface consists of the vertices in the atlas which are mostlikely to lie on the bone-cartilage interface (BCI) and which are nearcontact with an opposing bone. The non-contact surface is limited tothose vertices which are likely to contain cartilage, but which are notin contact with a bone. These surfaces can be seen in FIG. 20.

The profiles for each the contact surface are calculated along a pathbetween the current bone's vertices and the nearest vertex on thecontact bone. The profiles for the non-contact surface are calculatedalong the normal direction of the bone surface up to 1 cm. The localmaxima and minima for each profile are calculated and the profiles areplaced into one of three distinct classes. The mean profile for eachclass is shown in FIG. 21. If the profile contains a single maximum, itbelongs to class 1. These are the shortest profiles and correspond tolocations where the tibial and femoral cartilage are in close contactand are indistinguishable from one another. Profiles containing 2 maximaand one minimum are said to belong to class 2. These correspond toprofiles of intermediate lengths where there is a clear space betweenfemoral and tibial cartilage. Class 3 profiles are the longest profiles,where the femoral cartilage is usually well represented but the curvesin class 3 vary widely and are often irregular.

Any vertex having a profile belonging to class 1 or class 2 canimmediately be classified as belonging to the BCI. Class 3 profiles areadded or subtracted from the BCI based if the intensity level is nearthat of other BCI points and the likelihood that the point belongs tothe BCI, which is determined from the probability map of BCI.

After the BCI has been automatically determined the user is presentedwith the option for manual confirmation, and can edit the BCI using anumber of tools similar to those found in the bone segmentation editor.When the BCI is determined to be sufficiently accurate the segmentationof the cartilage model proceeds. The cartilage is segmented along theprofile dimensions using gradient information from the scan volumecoupled with the a priori knowledge of the cartilage thickness. This apriori knowledge provides an initial estimate of the likely cartilageedge, which is then updated by seeking the local maximum of the absolutevalue of the profile gradient. For class 1 profiles, the cartilage edgeis considered to be at the local maximum. For class 2 and 3 profiles,the local maximum of the gradient is used.

The cartilage segmentation can then be interactively adjusted ifnecessary before the final cartilage model is output. Segmented femoralcartilage can be seen in FIG. 22.

X-Ray bone reconstruction process is outline in FIG. 23.

X-Ray Images are taken using either traditional fluoroscopy orradiography. The number of images can be either one or more. Theimage(s) projection view is taken to maximize the obtainable informationby scanning at large angle differences. Accuracy of the system isdirectly proportional to the number of images, while speed is inverselyproportional. Radiographic scene properties focal length and imageresolution (of camera digitizer or film scanner) are manually input tothe system if not readily available in the image's file header.

The goal of preprocessing is to enhance the input images by reducingimage noise, increasing image contrast, and preparing the images forfurther analysis. This would be automatically done, with the possibilityof manual intervention for extreme image distortions. Gaussian, medianand Sobel filters will be used.

Calibration involves the extraction of the imaged bone pose within theradiographic scene. Before image acquisition, a calibration target isattached to the patient's leg [FIG. 24]. This target would contain radioopaque beads that would appear on the acquired images[FIG. 25,26]. Thebead projections would be automatically extracted from the images andused to roughly estimate the bone pose relative to the x-ray source.

The markers can then be automatically detected in the images usingmorphological operations. The placement of the markers on the subject ischosen to cover as large an image area as possible in all the frames.Enough markers were used to allow accurate pose estimation even when thefeature detection algorithm missed some of them, or when some of themwere outside the field of view.

By thresholding at various levels and removing (with morphologicaloperations) any objects that contain lines longer than the beaddiameter, we can isolate these beads. We find the centroids of the beadsby finding the connected components and then determining the centroid ofeach component. The calibration target is designed to minimize possibleoverlap of bead projections, which maximizes the number of detectedbeads.

The pose of the sensor is computed by finding correct associations (orcorrespondences) between the detected bead locations in the image andthe 3D bead locations. This is accomplished using an interpretation treesearch method. In this method, each level of the tree represents thecorrespondences between an image point and all possible model points,and each path from the root to a leaf node in the complete treerepresents one possible correspondence. When a search descending thetree reaches a node at which at least four correspondences have beenhypothesized, the correspondences are used to calculate a pose solution.In general, we only need three points to solve for a pose, but there maybe more than one solution since three points are always co-planner. Werequire that at least four points be used.

Once we have four correspondences we can calculate a pose. This posesolution is used to project the object points back onto the image forcomparison to the detected markers as a measure of how well thissolution fits the complete data set. If the pose solution fits the datawell then the correspondence and computed pose are correct. If the posesolution does not fit the data, then the correspondence is wrong, andthe tree is traversed further to check additional correspondences. Sincefor a large number of points this tree may be very large, we do notsearch the entire tree. In fact, we search the tree further only if thecurrent correspondence produces an error larger than some threshold. Ifit is below, we assume we have found a correct correspondence.

Once the correct correspondences have been found, we compute the poseusing a non-linear least squares method. Specifically, given an imagepoint P, a 3D model point Q, and the six-vector containing the correctpose parameters β, let the transformation to project onto the image begiven by the function ƒ(β, Q) such that P=ƒ(β, Q). The vector function ƒalso represents a perspective projection (whose parameters are knownfrom calibration). Linearizing around a current pose solution β we get

${\Delta \; P} = {\left( \frac{\partial f}{\partial\beta} \right){{\Delta\beta}.}}$

Given at least four point correspondences, we can solve for thecorrection term Δβ. This process is iterated until the solutionconverges. The final solution is the pose β that minimizes the squaredEuclidean distance between the observed image points and the projectedmodel points on the image plane.

The bone image segmentation block takes the preprocessed images asinput. Its goal it to extract the actual bone contours of the all theimages. This is done automatically using a statistical atlas of bonecontours generated using our database of 3D bones. A template averagebone contour from the statistical atlas is initially placed within theimage, and then translated and rotated to align with the bone's image.After that, contour deformation is statistically done to fit the targetbones image based on the image's intensity values and edges obtainedfrom the preprocessing step. Manual and semi-automatic contour editingtools are also available for the verification of the automatic process[FIG. 27].

The feature extraction module is responsible for the extraction of imageparameters from the images. These parameters are extracted from both thepreprocessed and segmented versions of the images. Types for featuresinclude information extracted from the segmented bone contour(curvature, geometry, aspect ratio . . . etc) or from the pixelintensity values (texture features such as Haralick and Tamura's texturefeatures).

This process is required because the calibration block is expected tointroduce an error due to the relative transformation between thecalibration target and the actual bone.In the case of fluoroscopy, the number of bone images is usually largeand the bone pose difference between coherent images is usually verysmall. Therefore, we will apply pose tracking using particle filters.This approach has been found to be useful in dealing in applicationswhere the state vector is complex and the images contain a great deal ofclutter. The basic idea is to represent the posterior probability by aweighted sampling of states, or particles. Given enough samples, evenvery complex probability distributions can be represented.As measurements are taken, the weights of the particles are adjustedusing the likelihood model, using the equation:

w _(j) ^(i) =P(y _(i) |x _(i))w _(j)

where w_(j) is the weight of the j^(th) particle.

The principal advantage of this representation is that it can representmultiple peaks in the posterior probability distribution, given enoughparticles [FIG. 28].

As measurements are obtained, the tracking algorithm adjusts theweights, and when enough data is obtained to disambiguate the states,eventually one of the particles has a high weight and all the othershave very small weights. Another advantage of this approach is that itcan determine when a unique solution has been obtained.

Resampling the particles. It is important to make sure that the samplingof states adequately represents the posterior probability distribution.In particular, we want a number of the samples to be close to peaks,where the weights are large. To ensure this, at each step we willresample the probability distribution by generating additional particlesnear large peaks, and discarding particles with very small weights. Wewill use importance sampling to inject samples based on bottom-upinformation.

Design of the likelihood function. It is important to design thelikelihood function so that it is as smooth as possible. If it had manynarrow peaks, some of those peaks could be missed unless a very largenumber of samples is used. The key is to use a similarity measure thathas broad support for each hypothesis. In other words, the similaritymeasure should gradually grow as our hypothesized state gets closer andcloser to the correct state, and not increase suddenly.

In the case of radiography, the number of images is usually limited,therefore, the features would be used in a Bayesian network frameworkwhere the expected output in the bone's pose given the current set ofimage features. This method can also be used to initialize the particlefilter in case of fluoroscopy. The Bayesian networks would beconstructed as directed acyclic graphs, where each node represents animage feature and node connections represent the conditionaldependencies. The output is the pose with the highest probability, giventhe set of input image features. The network output probability is basedon the probabilistic chain rule

${p\left( {x_{1},x_{2},x_{3},\ldots \mspace{14mu},x_{n}} \right)} = {\prod\limits_{m}^{n}\; {P\left( {\left. x_{m} \middle| x_{m + 1} \right.,x_{m + 2},\ldots \mspace{14mu},x_{n}} \right)}}$

Where x₁ x_(n) represent the image features and bone poses variables.

We use a 3D reconstruction algorithm based on GPU rendering simulation.The input for our method is the set of segmented images and theircorresponding bone poses. The number of images and the variety ofprojection poses indicates the amount of information that can beobtained about the bone shape, hence the accuracy of the output. Foreach of the input images, a graphical rebuild of the radiological sceneused to take the image is done. The x-ray source is represented by aperspective camera setup to simulate the radiological beam divergence.Within the camera's field of view, a template bone model is placed at apose mimicking the actual bone's pose within the radiological scene[FIG.29].

Having setup the graphical scene, bone projection images aresynthesized, whose contours are mapped to 3D points on the templatebone's surface by utilizing depth mapping rendering capabilities [FIG.30]. These 3D points are then systematically translated in space toeliminate the 2D contour error between the synthesized image and theoriginal radiographic image. As a result, using contour data from allimages, a cloud of 3D points that would produce the bone projectionssimilar to those of the input x-ray images will be produced [FIG. 31].

This transforms the problem to a 3D to 3D optimization problem. We willuse POCS (alternating Projection On Convex Hulls) method to quickly anduniquely find the best shape that is consistent with both thestatistical atlas as well as the generated point cloud. POCS is apowerful tool that has been used successfully for many signal and imagerestoration and synthesis problems. It is particularly useful inill-posed problems, where the problem can be regularized by imposingpossibly nonlinear convex constraints on the solution set. Iterativelyprojecting onto these convex sets results in a solution that isconsistent with all the desired properties. A short description of themethod follows.

In a vector space, a set p is convex if and only if for ∀xερ and yερ,then λx+(1−λ)y ερ∀0≦λ≦1. In other words, the line segment connecting xand y is totally subsumed in ρ. If any portion of the chord connectingtwo points lies outside of the set, the set is not convex A projectiononto a convex set is defined as follows. For every closed convex set, ρ,and every vector, x, in a Hilbert space, there is a unique vector in ρ,closest to x. This vector, denoted PCx, is the projection of x onto ρ.The most useful aspect of POCS is that, given two or more convex setswith nonempty intersection, alternately projecting among the sets willconverge to a point included in the intersection.

If two convex sets do not intersect, convergence is to a limit cyclethat is a mean square solution to the problem. Specifically, the cycleis between points in each set that are closest in the mean-square senseto the other set (FIG. 9).

In our method, we have two convex sets: 1. The set of all bones that canbelong to the statistical bone atlas. 2. The set of all bones that haveconstrained values for a selected number of vertices (the other verticescan have any values). The selected vertices are those for which we seecorresponding points on the image contour. We can project a bone vectoronto the second set by simply replacing each of the selected verticeswith the corresponding estimated point.

By alternating the projection onto one convex set and then the other, wewill quickly converge to a solution that is compatible with both sets.

Having obtained a patient-specific bone model, cartilage should be addedto complete the fitting surface where the jig should fit. Using ourdatabase of x-ray images, and their corresponding MRI scans, a Bayesiannetwork was created that identifies the level of cartilage deteriorationfrom the knee gap distance profiles of the x-ray images. Suchinformation is used as an input to the cartilage generation module.

System includes a smart database that stores patient informationfollowing HIPPA regulations. Data from different imaging modalities willbe attached with each patient including DICOM from MRI/CT, X-ray images,and ultrasound. Reconstructed bones and cartilage are stored in thedatabase. Virtual templating data including calculated landmarks, axes,implant sizing, and placement information are also stored in thedatabase. This component implements both relational and XML schema toprovide a powerful tool for data storage and manipulation.

FIG. 32 outline the process of creating a prediction model forreconstructing the articulating cartilage given the femur, tibia andpatella bone. To build this model we utilized a database of 2000 MRIscans. Bones and articulating cartilage were first segmented from thesescans. Bones were added to our statistical atlas to achieve pointcorrespondence and calculate the probability of bone cartilage interfaceareas on each bone. Bone to bone distances were calculated across thebone cartilage interface areas by finding the closest distance betweenthe two bone surfaces at each vertex. Cartilage thickness were alsocalculated at each of these location and used as a target to train aneural network and construct a Bayesian belief network to predict thecartilage thickness. Input for these system included and weren't limitedto the bone to bone distance, degenerative and deformity classificationof the knee joint, and measurement of the yams and valgus angle. Outputfor this is a prediction system that's capable of constructing cartilagein CT, X-Ray, US, Microwave and to guide cartilage segmentation in MRI[FIG. 33]. FIG. 34 shows process of identifying cartilage interface inMRI training datasets, whereas FIG. 35 shows the average cartilage map.FIG. 36 shows output for the prediction model for one of the test cases.

Microwave imaging system is comprised of an array where each elementacts as both a transmitter and receiver. The system architecture isshown in FIG. 37 where a low noise system clock (clock crystal) triggersa baseband UWB pulse generator (for instance a step recovery diode (SRD)pulse generator). The baseband pulse is upconverted by a localoscillator (LO) via a double balanced wideband mixer. The upconvertedsignal is amplified and filtered. Finally, the signal is transmitted viaa directional microwave antenna. The signal is received at all of theother antennas in the array and is filtered, amplified, downconverted,and low-pass filtered. Next, a sub-sampling mixer triggered by the samelow noise system clock is used to time extend the pulse by1000-100,000×. This effectively reduces the bandwidth of the UWB pulseand allows sampling by a conventional analog-to-digital converter (ADC).Finally, custom digital signal processing algorithms are used forbeamforming and creating the final cross sectional image, as shown inFIG. 38. A near-field delay and sum beamformer is used to recover theimage. The target scattered signals received by different Rx antennasare equalized in magnitude to compensate for different scatteringratios, propagation losses, and attenuations. Different phase delays ofthe Rx signals are used for beam steering. The interfaces betweenvarious tissue types is detected (air-skin, fat-muscle, cartilage-bone,etc.), as shown in FIG. 15. The experimental setup showing the UWBantenna array, the bones of the knee, and muscles surrounding the kneeis shown in FIG. 39.

The resultant received signals are used to detect tissue interfaces(i.e. cartilage-bone, fat-muscle, air-skin) from various angles and canalso be turned into 2-D cross sectional images. The tissue interfacedata is added to existing bone and cartilage atlas information for thepatient. This results in an extended feature vector to include the UWBimaging data. This process is outlined in FIG. 40. The tracked UWBantenna array is moved along the knee and multiple cross sectionalimages are obtained. The cross sectional images are registered togetherand allow a full 3-D analysis of the various tissue interfaces (withemphasis given to the cartilage-bone interface). Information related tothese tissue interfaces is added to the feature vector for the patient.This results in additional tissue interface information (i.e.cartilage-bone, soft tissue-cartilage) to be used in bone and cartilageatlas creation and various automated measurements pertaining to thearticular cartilage and bones of the knee. Finally, this information canbe included in the Bayesian estimation processes for the bone andcartilage atlases.

The system extends a diagnostic B-mode ultrasound machine to add to itthe capability of creating patient specific 3D models of the joint bonesand cartilage (For example the knee). A localization probe (optical orelectromagnetic) is rigidly attached to the ultrasound probe to trackits motion while the operator is scanning the joint (for example theknee). The transformation between the motion tracking probe's coordinateframe and the ultrasound probe coordinate frame is determined by acalibration process. The motion tracking probe provides the position(translation) and orientation of each acquired B-mode image (frame) sothe acquired images are registered in the 3D Cartesian space as shown inFIG. 41. The set of acquired ultrasound images along with their acquiredpositions and orientations are then used to reconstruct a volume of thescanned anatomy (similar to the volume reconstructed from CT or MRI)

Three or more alignment landmarks (predefined landmarks, like mostprotruding points on the femoral epicondyles) are then acquired using atracked A-mode probe or the B-mode probe, and then the mean model of thebone's (the bone to be modeled, for example the femur) atlas isregistered with the reconstructed volume using the acquired alignmentlandmarks using paired points registration. Then the proposed automaticsegmentation used for CT and MRI is applied to the reconstructedultrasound volume, so result in a segmented bone model as shown in FIG.42.

Flow chart explaining the ultrasound reconstruction system is shown inFIG. 43.

Virtual templating provides the ability to perform implant sizing,placement, and visualization on selected patient bones. Landmarks areautomatically calculated with high accuracy and repeatability. Implantsizing is performed along with implant placement. Users then may selectparticular implants and implant families on which to perform thesefunctions. Users may select from predefined or user-defined surgicaltechniques for placing the implant components and users may define newsurgical techniques for placement of both femoral and tibial components.For example, based on landmarks and axes, users may visualizeresections, full intact bones, and/or implants placed on resected bones.For example, in an exemplary embodiment, users may be provided withthree 2D orthogonal views and one 3D view for visualization and implantmanipulation. Users may modify implant size, family, and manufacturerinformation. Visualizations may include axes and landmarks overlaid onbones. Fitting results may be saved to the smart database. A surgeon mayutilize the various capabilities described herein to perform virtualtemplating, implant placement, virtual resection, and implantmanipulation, thereby producing quantitative results in preoperativetemplating for the patient and implant alignment. This surgeon editionof the virtual resection runs through the interne utilizing 3Dtechnology in applets (Java3d). Surgeon can modify, accept or deny thetemplating results. A range of motion is performed to verify implantalignment and placement. The transformation of the placed implant isthen used to transform the cutting tools in same alignment as patient.These screw holes are then translated to the jig to translate thisplacement into the operating room. [FIG. 44]

FIGS. 45,46 outline the process of jig creation, this process utilizes atemplate jig that is placed on the average bone from our statisticalatlas. this jig is then resized to match the size of the patient femoralcondyles and tibial plateau. Both tibia and femur models are thenintersected with the patient specific 3D bone models to create a uniquepatient imprint on the inside of the jig that guarantees tight fitduring surgery.

FIG. 47 highlights different jig design philosophies for different levelof joint degeneracy. FIG. 48 shows output femoral and tibial jig fromthe system.

FIG. 49 shows editor for polishing and verifying the automaticallycreated jig.

FIG. 50 shows process of verifying the output jig by projecting it onthe same space as the patient volumetric data.

Following from the above description and invention summaries, it shouldbe apparent to those of ordinary skill in the art that, while themethods and apparatuses herein described constitute exemplaryembodiments of the present invention, the invention contained herein isnot limited to this precise embodiment and that changes may be made tosuch embodiments without departing from the scope of the invention asdefined by the claims. Additionally, it is to be understood that theinvention is defined by the claims and it is not intended that anylimitations or elements describing the exemplary embodiments set forthherein are to be incorporated into the interpretation of any claimelement unless such limitation or element is explicitly stated.Likewise, it is to be understood that it is not necessary to meet any orall of the identified advantages or objects of the invention disclosedherein in order to fall within the scope of any claims, since theinvention is defined by the claims and since inherent and/or unforeseenadvantages of the present invention may exist even though they may nothave been explicitly discussed herein.

1-6. (canceled)
 7. A reconstruction system for generating apatient-specific bone shell representative of at least a portion of apatient's bone in its current state comprising: an imaging module tocreate a plurality of 2D image slices of a portion of a patient'sanatomy, where each of the plurality of 2D image slices is orthogonal toan axis extending through the patient's anatomy, each of the pluralityof 2D image slices including a bone segment comprising an enclosedboundary corresponding to an exterior of the patient's bone; arecognition module to recognize the enclosed boundary of each bonesegment on the 2D image slices by using a template 3D image bone shellthat is not patient specific; a construction module to produce a 3Dimage bone shell of at least the portion of the patient's bone byassociating the plurality of 2D image slices of the patient's bone basedon the recognition of the enclosed boundary of each bone segment, wherethe 3D image bone shell depicts a current state of the patient's bone.8. The reconstruction system of claim 7, further comprising at least oneof magnetic resonance imaging apparatus and computed tomographyapparatus to produce the plurality of 2D image.
 9. The reconstructionsystem of claim 7, wherein the 3D image bone shell is representative ofat least one of a femur, a tibia, and a humerus.
 10. The reconstructionsystem of claim 7, further comprising a generation module to generate a3D image surgical jig to mate with said 3D image bone shell, where the3D image surgical jig includes topographical features customized to theexterior features of the 3D image bone shell.
 11. The reconstructionsystem of claim 10, wherein the generation module outputs an instructionfile for fabricating a surgical jig having tangible topographicalfeatures of the 3D image surgical jig.
 12. The reconstruction system ofclaim 7, wherein the recognition module recognizes an outline ofcartilage appearing in at least one said 2D image slice, and furtherwherein the construction module produces a 3D image cartilage shellrepresenting at least a portion of a patient's cartilage using theplurality of 2D image slices of the patient's anatomy based on therecognition of the outline of cartilage.
 13. The reconstruction systemof claim 12, wherein the recognition module recognizes the outline ofcartilage appearing in at least one said 2D image slice by determiningcartilage thickness using at least one of statistical shape atlases andBayesian inversion probabilities.
 14. The reconstruction system of claim7, wherein the recognition module enables interactive intervention forat least one of contour adding, subtracting and painting.